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ABSTRACT 


ePortfolios (EPs) have been used in a variety of applications ranging from undergraduate assess¬ 
ment to graduate student work showcases. We hypothesize that the flexible, individualized nature 
of EPs makes them suitable assessment tools for graduate engineering programs, which are likewise 
flexible and individualized. Our investigation resulted in a systematic design process that can be used 
to create EP systems for graduate programmatic assessment. The process was used to develop EP 
assessment systems for three different engineering graduate programs. Results indicate that (1) the 
end products, the EP assessment systems, were directly affected by the graduate programs’ culture 
and vision of graduate engineering assessment, and that (2) EPs are flexible based on departmen¬ 
tal or program needs. Therefore, we conclude that EPs are appropriate for assessment in graduate 
engineering programs and that a systematic process of development can be used. 
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INTRODUCTION 


Graduate programmatic assessment is a challenging subject because of the individualized na¬ 
ture of graduate education, which requires a flexible yet robust assessment strategy. Students in 
graduate education are paired with a faculty advisor who assists them in designing their unique 
educational experience. Having a system that evaluates these experiences, but is standard enough 
for programmatic assessment, is vital. With their inherently individualized and flexible nature, elec¬ 
tronic portfolios, or ePortfolios, (EPs) are one promising assessment tool for graduate programmatic 
assessment. EPs can contain a variety of artifacts and evidence, which accommodates the individual¬ 
ized nature of graduate school and allows for the collection of indirect and direct assessment data. 
Also, programs can align EPs with guidelines and requirements, which accommodates the need for 
a standardized assessment system. 

This paper takes an exploratory approach and describes a primarily theoretical investigation into 
the use of EPs for graduate programmatic assessment. Through a literature review, we examine the 
use of EPs as assessment tools, theories that support the use of EPs for graduate programmatic 
assessment, and theories that support our design process used to create our EP programmatic 
assessment systems. Based on our findings, we hypothesize that EPs are an appropriate method 
for graduate programmatic assessment when based on a design process that adheres to a set of 
requirements for good assessment. In the methods section, we describe our development of an EP 
programmatic assessment system design process, grounded in both assessment requirements and 
communities of practice theory. The EP programmatic assessment system (i.e., the product of the 
design process in this paper) should not be confused with designing an individual EP. The systematic 
design process discussed in this paper focuses on meeting client needs at the program level, not 
developing an individual EP at the student level. The design process discussed in this work involves 
client meetings, consultations with EP support staff, presentations to stakeholders, etc. which form 
of a set of systematic steps that produce an EP programmatic assessment system. 

Finally, we describe the results of a practical application of the design process to three distinct 
graduate engineering programs at a large, public, research university in the southeastern region of 
the United States. We apply the design process to multiple programs to determine if using a single, 
systematic design process still allows the flexibility needed in order to meet different programs’ assess¬ 
ment requirements. We illustrate that our design process creates EP assessment systems grounded in 
good assessment practices and individually tailored to programmatic needs. While this paper focuses 
on the theoretical findings and the design process, and does not evaluate the effectiveness of each 
of the EP systems, the results from the three case studies indicate that EPs can be a valuable and 
customizable tool for assessing graduate programs in a variety of engineering contexts. 
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LITERATURE REVIEW 

EPs are individualized, flexible systems that can be used for assessment purposes, and if the 
design process suggested in this research is followed, a unique and customized system can be de¬ 
veloped for various department cultures and needs. Although research that discusses EPs for use 
in graduate engineering assessment is limited, we present theories (e.g., reflective judgment and 
motivation theory) that support their implementation. Additionally, we ground the design process 
in both good assessment practices and communities of practice theory. 

Electronic Portfolios as Assessment Tools 

For the purposes of this project, an EP is defined as a digital collection space of individual stu¬ 
dents’ academic artifacts and authentic, valid, and reliable evidence (Carroll & Calvo, 2006), which 
are stored and cataloged in an organized manner along with reflections to emphasize knowledge, 
competencies and/or skill sets possessed by the creator. An EP allows the students engaging in the 
process of collection and reflection of artifacts to establish a “digital identity or persona” (Clark, 
2009, p. 29). EPs should include work that has been collected, selected, and reflected on by the 
student, which helps the author of the EP develop ownership of their learning products (Cole, Ryan, 
& Kick, 1995). The collection process is not random, but rather a purposeful and discerning selection 
of artifacts (Paulson, Paulson, & Meyer, 1991). For the programmatic assessment purposes similar 
to this research, it is important to balance student ownership and appropriate data. For example, 
guiding categories were given to students to focus on artifact selection, but the students ultimately 
had selection control. 

While portfolios began as a paper method for artists, designers, and architects to showcase their 
creative and artistic ability for clients, portfolio use has expanded to other disciplines such as medicine, 
education, literature, social sciences, and engineering via electronic media (Rayudu, Heinrich, & Bhat- 
tacharya, 2009). The idea of using portfolios as a method of assessing student progress in writing, for 
instance, had been suggested numerous times since the 1960s, with an increase in use and exposure 
in the 1980s and 1990s, aided by a number of publications detailing their use (Elbow & Belanoff, 1997). 
Following the release of ABET EC2000 accreditation criteria, there was an even further increase in 
the literature exploring portfolios for undergraduate engineering assessment (e.g., McGourty, 2002; 
McNair, Paretti, Wolfe, & Knott, 2006; Williams, 2002). More recently, portfolios have been used for 
assessment of individual effort in group projects (Kavanagh & Cokley, 2011). Murphy (1997) argues 
that a portfolio offers an individualized and personalized form of assessment, something lacking in a 
number of academic settings that gravitate towards test-taking, such as engineering. It follows that 
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EPs allow students to better understand what they have learned and the progress that they made 
over a period of time, making EPs a good choice for a wide range of academic assessment needs. 
It should be noted that EPs, while similar to their paper based counterparts, allow an individual to 
connect various components of their portfolio through links which allows for a more integrated 
expression of oneself. This integration or linking may be lost in a paper based version. 

Portfolios and their electronic implementations have been studied extensively through case 
studies in primary, secondary, and higher education (baccalaureate and graduate) programs from 
countries all over the world including China, New Zealand, Croatia, the European Union, Canada 
and the United States (Wang, 2008; Rayudu, Heinrich, & Bhattacharya, 2009; Balaban & Bubas, 
2010; Malita, 2009; Meyer, Abrami, Wade, Aslan, & Deault, 2010; Georgi & Swenson, 2002; McNair & 
Borrego, 2010; Colgate, McKenna, & Ankenman, 2004). There has also been recent interest in using 
EPs for graduate assessment (e.g., Cambridge, 2001; Cyr & Muth, 2006; McNair & Borrego, 2010). 
However, only a few studies specifically address graduate assessment in engineering (e.g., McNair & 
Borrego, 2010). EPs have been utilized in academic fields across the spectrum, a demonstration of 
the flexibility of their usage in diverse contexts (Cambridge, 2001). While each previous study had 
different research questions and goals, the common conclusion drawn from the studies cited above 
was that EPs have had a positive measurable impact on students and faculty engaged in using EPs 
in the academic setting. The process we propose can assist in developing EP programmatic assess¬ 
ment systems for departments to use in a variety of academic settings ranging from the program 
level to labs to individual students. 

EPs have been classified in a variety of ways and have served multiple purposes. Meyer et al. (2010) 
listed the purposes of EPs as “process, showcase, and assessment” of student work (p. 90). Metz 
and Albernhe-Giordan (2010) outlined four categories including “learning, presentation, evaluation, 
and personal development” (p. 3564). Balaban and Bubas (2010) asserted that there are three types 
of EPs: “assessment, development, and showcase” (p. 331). McNair and Borrego (2010) stated that 
if implemented correctly, EPs can concurrently enhance student’s “learning development, career 
preparation, and program evaluation” (p. 2). Assessment or evaluation is a commonly cited pur¬ 
pose. Although EPs have the potential to assist graduate students in their learning, metacognition, 
showcasing of technical and research skills, as well as future career development as a professional or 
professoriate (Louis & McNair, 2011), the work presented in this paper focuses on the development 
of an EP system to create EPs that can be used as an assessment tool, specifically with graduate 
engineering programs. 

While implementing EPs in the classroom setting can provide the students, faculty, academic 
departments, and overall institution with valuable benefits, there are also some drawbacks to EPs. 
First, in a case study by Rayudu, Heinrich, and Bhattacharya (2009), EPs were implemented with 
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computer science undergraduate students in a university in New Zealand. The authors cited low 
participation in their case study because of students’ absence of lifelong learning skills and the 
study’s voluntary participation from the students to use the EP system. They contend that in order 
for EPs to succeed at higher education institutions, there needs to be buy-in from all levels of ad¬ 
ministration and faculty and more emphasis on a well-structured support system. Meyer et al. (2010) 
agreed, believing that in order for EPs to be implemented in an effective manner “school leaders, 
teacher educators, and pedagogical support staff need to provide consistent positive support to 
teachers as they learn to teach with new technologies and work within the changing realities of 
their school environments” (p. 90). In a paper by Carroll and Calvo (2006), the authors discussed 
the need for reliability and validity in terms of providing credible artifacts chosen by the students 
to upload to their EP. The authors were concerned that the evidence could potentially be tampered 
with and present an unrealistic and untrue version of the student to prospective employers or other 
stakeholders. Finally, Kimball (2005) stated that sometimes EPs can be marketed as a “quick and 
painless process” only requiring “40 minutes of work,” which is a misrepresentation to students 
since developing a quality EP can take hours of collection and reflection (p. 435). Similar concerns 
were faced in this study and are discussed in later sections. 

Theories Supporting ePortfolio Use for Graduate Assessment 

EPs have had limited use in graduate education assessment to date, but we think that they are 
indeed an effective means of assessing graduate programs. The design process developed in this 
study provides a theoretically grounded framework of practices to assist engineering departments 
in the development of their own EP systems. Specifically, the theory of reflective judgment and mo¬ 
tivation theory help explain how EPs can support students taking ownership of their own learning 
processes, which is critical to graduate education. 

Reflective judgment 

An EP system, as suggested by Backlund, et al. (2001), can encourage students in higher edu¬ 
cation to reflect on the educational process, gather the individual ideas that they have obtained 
through coursework and research, and obtain a higher level understanding of what they have 
been doing. Condon (1997) argues that portfolios can help “build bridges” between the often dis¬ 
jointed ideas students encounter in higher education. From the faculty perspective, such a system 
could aid in the assessment of epistemological maturity. Previous research by Stone (1997) shows 
that portfolios are a useful tool for assessing early cognitive development, and Heiges (1997) 
argues for portfolio usage in place of a doctoral examination as the final step to awarding a PhD. 
Indeed, in a graduate level program where the goal is to produce master’s and doctoral 
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candidates with the ability to think and conduct research for themselves, EPs can play a valuable role in 
assessment. 

In learning and epistemic theory, intellectual maturity relates closely to the theory of reflective 
judgment. Reflective judgment is a model of how individuals’ epistemological assumptions (as¬ 
sumptions relating to the nature and organization of knowledge) relate to their understanding and 
justifications of concepts (King & Kitchener, 2004). At a low level of epistemological development, 
knowledge is viewed in terms of absolutes; something is either right or wrong. As the complex¬ 
ity of how a person views knowledge increases, so does the uncertainty with which they apply it. 
At a moderate level of epistemological development, one might view all knowledge as contextual 
within the situation to which it pertains, leading to the possibility of unsubstantiated arguments 
when applying said knowledge. At the highest level of development, one is able to navigate such 
uncertainties with reason and construct probable solutions to problems. 

The implication of this theory is that, by the time a student reaches graduate school, he or she 
should be able to demonstrate a mature epistemological foundation. In Hoey’s (2008) overview of 
graduate level assessment in engineering education, we see this view reflected in the suggested 
“professional and attitudinal skills” that graduate students should be able to demonstrate, especially 
in: (a) critical and reflective thinking skills, (b) ability to apply theory to professional practice, and 
(c) ability to conduct independent research. The customizable nature of EPs, focused on evidence- 
based critical reflection and application to professional standards, has the potential to engender and 
assess a critically necessary and epistemologically mature skill set in graduate education. Addition¬ 
ally, programmatic requirements, such as those referenced in Hoey’s (2008) “Principles of Graduate 
Program Assessment,” can be appropriately addressed through EPs situated in the individualized 
contexts of graduate programs. 

Motivation theory 

The benefits of an EP system can also be examined through motivation theory. Motivation con¬ 
cerns the analysis of the factors that cause a person to participate in an activity, especially related 
to learning. As discussed in Eccles and Wigfield’s (2002) overview of motivation theories, differ¬ 
ences in thought processes can lead to differences in ability to retain information. Learners who 
are extrinsically motivated, those who are greatly affected by external factors such as grades and 
parental pressure, are more likely to engage in surface learning, a shallow memorization of ideas 
leading to reduced long-term recall of knowledge. Intrinsically motivated learners, those who are 
motivated by personal interests and goals, are more likely to experience deep learning and greater 
long-term recall of knowledge. 
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By providing students with a means of contextualizing knowledge that they have learned from 
multiple sources, EPs can have a beneficial effect on motivation. When students reflectively pro¬ 
cess information, they are learning through the lens of their personal academic and career goals; 
this can increase their interest in such topics. Individual interest, according to Schiefele (1999), 
plays a significant role in how much students learn as well as how deeply they learn it. Additionally, 
increasing interest can also play an important role in student persistence in an activity. From the 
framework of Eccles et al.’s (1983) expectancy value theory, individual interest has the potential to 
increase the value of an activity in the following ways: (a) increasing how the activity is relevant to 
perpetuating one’s sense of self, described as “attainment value”; (b) increasing how enjoyable the 
activity is, described as “intrinsic value”; and (c) increasing how useful an activity is to one’s future 
goals, described as “utility value.” With an increase in these values comes an increased likelihood 
to persist and perform well in an activity. Thus, since EPs allow students to self-reflect and become 
directly involved in their education, the use of EPs for graduate assessment can be predicted to 
have a positive effect on student performance in graduate programs. 

Theories Supporting the Design Process 

The design process developed through this research was created to allow for a flexible system 
that can assist a variety of departments in developing EP systems for assessment. The process was 
created with good assessment practices as the foundational theory, but the process also employed 
the theory of a community of practice to ensure faculty and student buy-in which is critical for ef¬ 
ficient implementation. 

Good assessment 

Good assessment is not an outcome of the design process so much as it is consideration for 
assessment at the beginning of the design process, and it is the responsibility of all stakeholders 
(Borrego, 2008). Suskie (2008) summarizes good assessment as assessment that: (1) concentrates 
on and comes from clear and important objectives, (2) is cost effective in terms of money and, 
more importantly, time, (3) produces truthful and accurate results, (4) is utilized, and (5) is valued. 
According to Suskie, for assessment to be good assessment, all five of these items must be met. 
Finally, for programmatic assessment, good assessment practices require information to be fed back 
to the department (Qualters, Sheahan, Mason, Navick, & Dixon, 2008). Therefore, when designing 
assessment, stakeholders and designers should ensure the assessment includes Suskie’s five quali¬ 
ties plus feedback considerations from the very beginning. 

Though multiple authors give criteria for developing or evaluating an assessment plan, we 
use Suskie’s (2008) criteria to inform our design process because the criteria are concise and 
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comprehensive. Suskie’s criteria were developed as summary points of previous literature and 
guidelines. Other authors focus on specific details of plans, such as evaluating a plan’s methods 
(e.g., Spurlin, 2008), and others focus on a specific context, such as assessment within a single 
course (e.g., Raubenheimer, 2008). However, Suskie gives five overarching criteria that cover an 
assessment plan in its entirety. Also, Suskie’s good assessment criteria have been used in the 
engineering context so they are appropriate for this design process related to graduate student 
assessment in engineering. 

Communities of practice 

While originating in industry (Wenger, McDermott & Snyder, 2002; Brown & Duguid, 1988), the 
idea of a community of practice has been expanded to education and was used as a key component 
in our design process. As defined by Wenger (1998), a community of practice encompasses three 
main elements: (a) mutual engagement, (b) a joint enterprise, and (c) a shared repertoire. Mutual 
engagement refers to all parties in the community communicating and interacting with one another. 
Joint enterprise means that all participants are working toward a common goal. Finally, shared rep¬ 
ertoire refers to the body of knowledge available and used by the group. In engineering education, 
communities of practice have been used in multiple contexts ranging from learning in the classroom 
to virtual learning (e.g., Paretti, 2005; Paretti & McNair, 2008). The community of practice formed by 
the assessment design team members, stakeholders, and the EP development community influenced 
the EP design process in multiple stages and allowed for a stronger design process that considered 
ideas from a multitude of sources, thus laying a strong foundation for student and faculty buy-in. 

Synopsis of Literature 

As explained above, previous research demonstrates that EPs have been used in a wide variety 
of contexts. By giving students the ability to select artifacts and self-reflect, motivation to learn 
and succeed can be increased, and the use of EPs can be an individualized, flexible graduate as¬ 
sessment tool. Furthermore, our hypothesis that the individualized, flexible nature of EPs makes 
them suitable assessment tools for graduate engineering programs is also supported by literature. 
By incorporating the tenants of good assessment with a reflective judgment element in an EP de¬ 
sign process, we contend that EPs can be effective tools for graduate program assessment when a 
strong community of practice supports the design process at key steps. Including good assessment 
practices in the design process is necessary to ensure EP outcomes are consistent with the desired 
stakeholder outcomes. 

In the remaining sections of this paper, we detail the development of a design process meet¬ 
ing these guidelines. Furthermore, in order to examine the effects of applying a common design 
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process to multiple contexts, we apply the design process to create EP assessment systems for 
three different engineering graduate programs (a department in a traditional engineering discipline, 
a department in a developing engineering discipline, and an individual research lab). In designing 
multiple systems, we specifically address the research question: 

When a single EP design process is applied to multiple contexts in engineering, how do 
stakeholder needs continue to shape the EP assessment systems? 


METHODS 

To aid in designing EPs for programmatic assessment, nine graduate students in an assessment 
course were tasked with developing EP programmatic assessment systems for three unique clients. 
The clients were (1) a department for a traditional engineering discipline, (2) a department for a 
developing engineering discipline, and (3) a research laboratory. The clients consisted of depart¬ 
ment heads, graduate directors and coordinators, a laboratory director, and selected students from 
the programs who all came together to create a community of practice. The design team members 
were mostly engineering education PhD students, but a few members were majoring in other areas 
of graduate level engineering such as aerospace engineering and industrial systems engineering. 
The study produced two outcomes: (a) an EP design process that can be used by other depart¬ 
ments or clients to create EP assessment systems, and (b) three example EP systems for assessing 
graduate engineering programs. The EP design process was driven by Suskie’s (2008) principles of 
good assessment and influenced by the relevant community of practice. 

The Community 

All three of the design teams involved in this project were influenced in some way by the develop¬ 
ment of a community of practice that instantiated a process of learning by working with an array of 
EP participants and sharing experiences, information, and knowledge. To fully understand the impact 
of the community of practice, the community itself must be defined. For this research, the core com¬ 
munity included the Engineering EP design teams, but there were also key players that influenced the 
core community including the supervising faculty member, the project coordinator, the university EP 
support staff and the faculty and student clients. The core community was developed through the 
development of each design, but in a broader sense, the core community was part of the larger EP 
community at the university and was directly influenced (especially through research texts) by the 
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assessment community that exists in education in general. The use of EPs at the university has grown 
over the past years as more departments implement them in their programs. While participants in the 
project described here learned through practice with the larger community of practice, the work done 
by the core group also helped expand EP use at the university level, both by increasing awareness 
of EP flexibility and by broadening the range of implementation options. Additionally, the core com¬ 
munity of this research (the students, faculty, EP software team, etc.) helped to bring the engineering 
education assessment community and the general EP community closer together. 

In terms of Wenger’s (1998) definition of a community of practice, three main characteristics, 
mutual engagement, joint enterprise, and shared repertoire, must be met by any group for it to be 
considered a community of practice. For this research, there was “mutual engagement” in this com¬ 
munity because of presentations and critiques shared between design teams. While the specifics 
of each design were different, the overall goal of each team was similar: to create an assessment 
system that meets client needs while adhering to good assessment practices (Suskie, 2008). There 
was a “joint enterprise” to finish the projects and develop the best EP system possible to meet the 
clients’ needs. Finally, there was “shared repertoire” because as teams learned facets of the EP 
platform, these items were shared between groups to help improve the project for all participants. 
Also, before the study began, the participants were all exposed to similar bodies of knowledge that 
served as the base for the EP development. The supervising faculty member, the project coordina¬ 
tor, and the clients contributed to the shared repertoire during meetings. 

The notion of a community of practice was not initially the guiding factor of the project, but 
reflection on the experience showed that the process of developing the EP systems was indeed 
influenced by this community-centered approach to learning and working. The synergy created by 
the community of practice experience enabled the individual EPs to be improved. 

Design Process 

The design process was developed through interactions within the community and based on the 
literature presented in early sections. Each design team used the common design process to develop 
their programmatic assessment EP system. This process is illustrated in Figure 1. 

Each of the delineated design process steps are discussed below in detail and examples of how 
the design process was implemented are available in the case studies section. The steps in the pro¬ 
cess allowed the EP design teams to ensure all stakeholder objectives were met while maintaining 
standards of good assessment. 

1. Start: Initiate the project with high-level goals and outcomes. Ask questions such as “What is 
being assessed?” “What program goals are being supported?” “What are the desired student 
outcomes?” 
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2. Define project objectives: Define the specific measurable project goals for each design team 
based on the client’s needs, program objectives, current assessment practices etc. Each goal 
should support criteria for good assessment. 

3. Identify EP design methodology: Design methodologies can take many forms such as backward 
design (Wiggins, 1998) or simply working with what is already in place. To be successful with 
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developing an EP assessment system, a single methodology or a clearly focused methodology 
should be established to guide the team through the EP assessment system development. 

4. Identify stakeholders: Stakeholders may include the faculty and students of the client depart¬ 
ment, the university’s EP center, the supervising faculty member, the assessment design team 
members, and others. 

5. Review and refine stakeholder goals: Define the department goals to ensure alignment with 
good assessment, determine any reasonable additional goals within the design scope, and 
communicate those goals back to the stakeholders to ensure there is goal agreement. 

6. Develop a preliminary EP solution (prototype phase): Develop a preliminary solution including 
mapping an assessment technique for each goal, as applicable, and recommending an EP 
design encompassing the proposed solution for each goal. 

7. Document the prototype in a manner that can be tested and verified: Enlist the support 
of the stakeholders to ask the questions “Is the EP achievable?” and “Does the EP meet 
objectives?” 

8. Receive feedback from all stakeholders: The stakeholders provide feedback to the design 
teams. Feedback includes potential EP software or interface limitations as well as subjective 
comments from the stakeholders. Use the feedback to iterate, as schedule and scope permit, 
to a refined solution. 

9. Finalize and document the recommended EP: Deliver the prototype EP assessment tool and 
documentation using several different forms of media (e.g., word processing documents, 
presentations). 

10. Communicate results: Communicate the final EP recommendation to the wider audience of stake¬ 
holders including the end users of key faculty and students within the respective department. 

11 . Closeout design with reflection write-up: Reflect on the design process and assessment prac¬ 
tices to enhance future EP assessment system development. 

Design Process Execution 

This design process involves all elements of a good design process as defined by Sobek and Jain 
(2004). Within the context of this paper, a good design process 1) will be used by the clients, 2) 
contains good assessment, and 3) meets the clients’ objectives. The design process includes a feed¬ 
back loop to help ensure the final outcome was good. Objectives and requirements were given in the 
form of the needs of primary stakeholders for each design team. A plan was scoped, the schedule 
developed, and assignments were made (Design Process Step (DPS) 2 - 5). The plan was executed 
and “controlled” through iterations (DPS 6 - 9). The products were presented to the key stakeholders 
(DPS 10), and the projects were “closed” (DPS 11). The use of a systematic process to develop the 
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EP systems allowed the higher-level objectives to be met by placing more emphasis on meeting the 
expectations of the stakeholders and ensuring the EP recommendations contained good assessment 
practices opposed to focusing the design process on the mechanics of an individual EP. 

Good Assessment in the Design Process 

Each design team was required to follow good assessment practices during the development of 
their EP programmatic assessment system. Additionally, they were required to meet their specific 
client’s needs. By following the design process outline in this paper, each group was able to address 
their client’s needs while upholding the requirements of good assessment. Below, evidence is given 
to link the design process to Suskie’s (2008) five principles of good assessment. 

Suskie’s (2008) first requirement of good assessment is that assessment must concentrate on 
and come from clear and important objectives. Each of the design teams began the prototype phase 
with clear and important objectives (DPS 5 - 6). During the initial client meetings all of the design 
teams were concerned with defining learning objectives for their specific department. The teams 
started with the objectives the department already had in place for assessment purposes, which 
ranged from general regional assessment items to specific departmental goals, and then refined 
these objectives where possible to make them more concise. These steps ensured that the assess¬ 
ment system developed had a strong foundation of objectives to be built form. 

While creating the assessment items for the EP designs, each of the design teams were mindful of 
the time required to complete the tasks. Suskie (2008) defines another element of good assessment to 
be assessment that is cost effective in terms of money and, more importantly, time. Some clients had 
more flexibility in time compared to others, but the overall goal of each team was to create an efficient 
and timely system. Efficiency, in terms of time, was important to students, advisors, and the depart¬ 
ments as a whole (DPS 6 and 8). It should be noted that while time was important to all stakeholders, 
the appropriate amount of time to devote to the EP system varied across groups. This was something 
that the design times had to negotiate between the students, advisors, departments, etc. 

According to Suskie (2008), the information collected through an assessment system must be 
truthful and accurate (DPS 6, 7 and 9). Much of the information in an EP is self-reported, so much of 
the responsibility for truthfulness and accuracy falls to the student using the EP. However, question¬ 
naires were designed to solicit specific information with regards to degree progress and achieve¬ 
ment to help ensure truthfulness and accuracy through consistency. Also, the reflective practices 
of the assessment encourage honest responses, but the self-reported nature of the EPs will always 
be considered when reviewing data. 

Finally, feedback during the design process was the primary mechanism to ensure the clients 
valued the EP solution and to improve the likelihood of use (DPS 8). Suskie’s (2008) last two 
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requirements for good assessment are value and use. The original departmental assessment sys¬ 
tems were not electronic and direct feedback was rarely exchanged. By improving and increasing 
feedback, the EP systems will be used more frequently and valued by stakeholders because the 
system provides two-way communications between groups. 

Meeting client needs while adhering to a set of principles related to good assessment was not 
always simple or easy, but following a specific design process made merging the two requirement 
sets possible. 

EP Software Platform 

Though many EP software platforms exist (e.g., Google Sites, DropBox, Weebly), one of the major 
requirements of the case study implementations was that each team had to develop a system to be 
used with the university supported course management site - a portfolio application included with 
Sakai Collaborative Learning Environment version 2.6x. The EP system available through this pro¬ 
gram includes the typical webpage front-end (e.g., eResume) and an organizational matrix for data 
collection. The matrix consists of rows to collect artifacts and columns for each year of a student’s 
program. Example matrices are included in Appendix A and discussed in the case study sections. 

There were both benefits and drawbacks to using the university provided interface. One of the ben¬ 
efits was that the site would be secure, and students could easily store and access past coursework to 
use in their EP. Another benefit was that design teams did not need to be familiar with programming in 
any way to use the system. The matrix contains artifact selection guidelines/categories to encourage a 
purposeful collection of artifacts. The course management site’s EP capabilities, along with the university 
EP support staff, make constructing the EP user friendly and self-explanatory. A limitation was that the 
course management site was fairly new, so certain features, such as data collection capabilities, were 
still being developed. This limitation did not hinder the design teams from developing the EP program¬ 
matic assessment systems, but it did require them to consider multiple solutions to their clients’ needs. 
These benefits and limitations of the platform required for the EP project greatly influenced the systems 
that were designed, but because each group was required to use the same platform, it streamlined the 
design process. The EP design process is applicable regardless of EP platform used. 


RESULTS OF DESIGN PROCESS APPLICATION: THREE CASE STUDIES 


The EP design process was used to create EP assessment systems for three different graduate 
engineering programs: Case 1, a department in a traditional engineering discipline with approxi¬ 
mately 30 Master’s students and 50 PhD students; Case 2, a department in a developing engineering 
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Figure 2. Example portfolio snapshots. 


discipline with about 30 Master’s students and 30 PhD students; and Case 3, an individual research 
lab with 6 Master’s students and 8 PhD students. The case studies presented below are tied back 
to the design process steps (DPS). The case study results are summarized in Appendix B. The mo¬ 
tivation for implementing EPs differed, as well as the level of buy-in from clients, i.e., both faculty 
and students. Even so, for all three cases, use of the end-product assessment system by students 
was projected to become a program-level requirement upon completion of this project. The three 
cases and how stakeholder interaction, needs, and feedback shaped the EP assessment systems 
are described below along with a discussion of how the end-products were shaped by the gradu¬ 
ate programs’ culture and vision of graduate assessment. To aid the discussion, example portfolio 
snapshots are shown in Figure 2. Students can create a professional portfolio (left) for sharing with 
the public (e.g., potential employers). The professional portfolio is based off data entered into the 
assessment system (right). The assessment system uses artifact collection matrices (Appendix A) 
to give advisors a quick snapshot of students’ degree progress. (For more information see: |http://1 
ltinyurl.com/EP-toolsl) 

Case 1: Traditional Engineering Department 

At the university-level, the graduate school collects data for the Southern Association of 
Colleges and Schools (SACS) and, thus, the college of engineering has developed four objectives 
for that purpose. These objectives stipulate that graduate students must demonstrate: (a) technical 
competence, (b) awareness of technical literature, (c) research ability, and (d) communication skills. 
The motivation for using EPs in the traditional department stemmed from a desire to streamline data 
collection for these four broad objectives while allowing students the flexibility to enhance their EPs 
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(DPS 2). The current assessment and reporting system requires students to fill in a hard-copy form, 
obtain appropriate signatures, and submit the form to the departmental office. The department 
desired to use EPs to replace the current system. 

During the design phase (DPS 5), the department chose streamlining SACS objectives over developing 
specific departmental outcomes. However, the department was interested in the idea of self-reflection, 
so optional reflection components were incorporated into the system (DPS 6). The artifact collection 
matrix is shown in the Appendix A (DPS 7). The Case 1 matrix is shaped primarily by the four broad out¬ 
comes, with the additional requirement to archive advisor feedback and supply student reflections (DPS 
6). Under the current system, many students do not receive the feedback their advisors give on their 
annual reports directly. In the new system, the matrix will store a copy of the advisor’s feedback form in 
addition to prompting both the advisor and student to schedule a meeting to discuss the form. 

The department values the EP assessment system (DPS 8-9) because the system provides data 
for accreditation, funding, and general reporting. The system improves the current data collection 
process by removing the need to transfer data from paper forms to electronic spreadsheets, and 
it can collect rich data in the form of student papers, graduate committee feedback, and optional 
student reflections that were not directly available to faculty and the department previously. 

For the students (DPS 8-9), the value is less obvious right now in the initial stages. The EP 
design team struggled with finding apparent value for the students in this department because 
of the department’s traditional culture, which emphasized technical skills and deemphasized 
“soft-skills”. Students’ impressions were that creating an EP was more work and possibly more 
time consuming than the previous system. The students struggled with the promise that a little 
extra time can greatly improve quality. This concern is valid considering the great amount of high 
caliber work required of students in the program currently. The value of EP assessment appears 
to be access to an online EP or eResume (e.g., McNair & Borrego, 2010). However, the online 
eResume portion of the system is limited and many students were interested in using other 
platforms (e.g., Google Sites) or even designing their own webpages. Since the students in this 
program preferred other methods of accomplishing these tasks, the value is difficult to directly 
identify. Once the system is used, it is anticipated that students will value the improved system 
for receiving feedback from their advisors and the ability to review and self-assess their progress 
through the degree program. 

Case 2: Developing Engineering Department 

In addition to a streamlined system for many departmental administrative requirements, the 
newer, developing department’s faculty desired a way to assess graduate students’ understand¬ 
ing and application of the research process (DPS 2). Faculty and students (DPS 4) wanted the 
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system to be as transparent as possible, meaning students need to understand the goals of the 
assessment technique and why it is being applied (DPS 5). Faculty and students also wanted the 
EP to accurately reflect student progress through each graduate program (PhD and Master’s) 
(DPS 5). At the program level, artifacts from the EP will provide objective evidence of meeting 
program requirements for accreditation purposes. 

To assess these outcomes (DPS 6-7), a multifaceted approach for the EP system based on student 
reflection was developed. The introductory page to their portfolio requires students to elaborate 
upon their reasons for entering graduate school and choosing their career path. Subsequent pages 
ask students to demonstrate knowledge of the research process by elaborating on their research 
goals, reflecting upon important research accomplishments, and setting and reflecting on semester 
and annual goals. Students are also required to upload any research publications as another ex¬ 
ample of research accomplishment. As students progress through their academic career, their EP 
should begin to look like an overview of the research process in action, with reflections and goals 
encouraging them to remain self-aware of their progress through the program. Research goals are 
to be revised throughout a graduate student’s career, resulting in a final EP that reads as an in- 
depth exploration of their research career. Through these responses, faculty will gain insight into 
how well students grasp the research process, and students will be able to construct an overview 
of what they have accomplished and what it means with respect to their goals. In conjunction with 
an online plan of study and an overview of research publications, this system allows students and 
faculty to easily gauge progress towards a graduate degree. The artifact collection matrix is shown 
in the Appendix A. The Case 2 matrix is shaped by stakeholder goals, which are then mapped to 
the four broad, graduate school outcomes. 

Departmental buy-in (DPS 8-9) was strong and students were directly involved in the design 
process. The department values EPs because they believe the system will improve graduate educa¬ 
tion within the department. The students (DPS 8-9) who participated in the design phase also had 
strong buy-in for the EP components. However, other students (DPS 8-9) were more cautious and 
sought to determine EP value; the students are leery of anything that appears to be more work and 
require more time. Students desired to improve faculty feedback, including prompting the profes¬ 
sors when feedback was due in order to save time. 

Case 3: Engineering Research Laboratory 

The individual research laboratory is situated within a large, traditional engineering department. 
However, the department is not the same department as Case 1, and does not currently use EPs. 
Rather than streamline data collection, the motivation to use EPs within the laboratory stemmed from 
increasing student-advisor feedback throughout a student’s degree program to decrease degree 
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completion time (DPS 2). The belief is that consistent and constant feedback will keep students on 
track, resulting in faster matriculation rates. Similar to Case 2, faculty and students wanted (DPS 
4) a transparent system with clear goals (DPS 5). It was important for students to know why they 
were collecting particular artifacts. 

When the assessment design team began working with the laboratory director, over 30 specific 
artifacts had already been identified for inclusion in the EP. Many of the artifacts were related to 
degree completion (e.g., plan of study, qualifying exam), or were related to specific laboratory re¬ 
quirements (e.g., safety training, papers submitted). Most of the artifacts involved documentation 
or items that were previously created so no new work was required for the EP. Rather than present 
students with an overwhelming list of artifacts to collect for each year, the design team tied spe¬ 
cific artifacts back to the four broad graduate level outcomes, a laboratory requirement category, 
and a miscellaneous category (DPS 6). Then, for each year, students would need to complete or 
collect artifacts in each of the six overarching categories, but not the entire list of 30+ artifacts. 
Self-reflection was not a priority for the laboratory faculty or students. Reflections were viewed 
as additional work with little benefit. To minimize cost in terms of time, but still allow for reflec¬ 
tions, reflection prompts were limited to the overarching categories. In other words, students were 
asked to reflect on their overall technical competence, their overall communication skills, and so on. 
The artifact collection matrix is shown in Appendix A (DPS 7). Individual artifacts shape the Case 
3 matrix. The artifacts were mapped to six categories to provide EP goal clarity and tie students’ 
work directly to graduate school outcomes. 

Laboratory buy-in (DPS 8-9) was strong. A unique feature of working with a small laboratory is 
that the faculty leader has close ties with the students and was a persuasive and credible resource 
when building student buy-in. The faculty leader valued EPs for managing individual students’ 
programs. He was able to provide concrete examples of how students would benefit, such as 
being aware of deficiencies and more timely graduation rates. Also, though only a few students 
participated in the design phase, the percentage of participation was higher than either of the 
other cases. 


DISCUSSION 

The design process and case studies discussed in this paper demonstrate the flexibility of EPs for 
graduate assessment framed by five tenants of good assessment (Suskie, 2008). The single design 
process used demonstrated that multiple products can be developed to meet unique client needs 
while following a set of predefined requirements. As outlined previously, the idea of a community of 
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practice (Wenger, 1998) also influenced the design process by providing a space for collaboration 
and community-centered development, both of which are needed for successful EP development 
for departmental assessment purposes. 

The resulting product from each design was able to meet department or laboratory needs. Addi¬ 
tionally, EPs were used as a method for students to demonstrate epistemic maturity. That is, through 
optional and required reflections on the research process, the EPs designed encourage students to 
take steps towards reflective judgment (King & Kitchener, 2004) by considering how their research 
and academic experiences fit in with their research and career goals. Providing a means for gradu¬ 
ate students to reflect on their individualized graduate experiences further supports our claim that 
EPs are an appropriate system for graduate assessment in engineering even though they have only 
been used on a few occasions (e.g., McNair & Borrego, 2010). Additionally, the process of reflection 
encourages self-assessment and helps students self-identify the areas in which they must improve. 
Such demonstrations of reflective thinking are necessary, according to Hoey (2008), for proper 
graduate assessment. 

The process of developing an EP also serves to motivate students in their graduate career. 
Reflecting on what they have done and what they need to do enables students to consider why 
certain experiences are important to them. By helping students incorporate necessary activi¬ 
ties in a graduate program into their activities of interest, EPs can promote intrinsic motivation, 
deeper learning, and improved student retention (Eccles and Wigfield, 2002). All of these items 
support the use of EPs in graduate assessment. The resulting products also made apparent levels 
of epistemic and cognitive maturity, creating an avenue for departmental and advisor feedback. 
Suskie (2008) identifies use and value as key components to good assessment. The feedback in¬ 
corporated into the EPs helps to strengthen the value and use of EPs for all stakeholders involved 
in the assessment. It may also provide another avenue to increase motivation through consistent 
feedback and verification. 

From a departmental perspective, the EP assessment systems designed in this study allow for 
the collection of data for their assessment processes in a multitude of different ways. This is demon¬ 
strated through the variety of matrices, surveys, and forms created through this project. As students 
develop their EPs throughout their graduate education, committee members and advisors can easily 
track progress and development over such a time span. The EPs developed help to streamline re¬ 
view processes by making vital data available in one central, easily accessible location for interested 
parties and stakeholders. Collecting such data provides information that can be used for accredita¬ 
tion purposes. It should be noted that graduate engineering programs are not accredited through 
ABET, but do have review boards based on their geographical location (for this project that would 
be SACS) so collecting data is still a very important component at the graduate level. 
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The EP systems developed herein also promote the accomplishment of other factors important to 
graduate assessment in engineering programs. When once again compared to Hoey’s (2008) skills 
essential to graduate-level study, the designed EP products are a place for students to demonstrate 
their ability to conduct research, to advance knowledge within their discipline, and to reveal high- 
level written communication skills. Thus, the results of this project further demonstrate that EPs 
are a powerful system for engineering graduate assessment that can be customized to individual 
department or group needs. 


CONCLUSION 

While the three case studies revealed a plethora of information about the development process 
for creating an EP and the variety of things it can measure, there were two general trends that 
emerged from this work. 

1. A standard systematic design process based in good assessment can be used to design EPs 
for graduate programmatic assessment. 

2. Stakeholder needs continue to shape EP systems, even when a single design process is used. 

These two overarching ideas encompass many details related to EP work that were showcased 

through the systems created by each of the three EP development teams. The EP assessment systems 
developed were effective and appropriate for graduate program assessment because they were all 
based on specific departmental needs. For example, Case 1 demonstrates a system focused on 
broad graduate program goals, Case 2 showcases a system developed for unique departmental 
goals, and Case 3 exhibits how EPs can be developed to achieve general assessment goals through 
the collection of distinct artifacts specifically related to a research lab. These are just three cases 
that illustrate how EPs can be used effectively and appropriately for assessment purposes. The 
individualized departmental needs and cultures affected each EP in a different way. For example, 
some clients were more concerned with timeliness and efficiency while others were focused on 
graduate student reflection, progress, and overall participation. Each client had a distinct culture, 
but by using a standardized design process, an effective EP was developed that would function 
natively in each context. 

The ability to accomplish many essential graduate-level assessment needs combined with the 
flexibility to fulfill individual client needs is the major strength of the design process outlined in this 
study. By taking steps to ensure that (1) the elements of good assessment are present and (2) the 
needs of all stakeholders are considered/built-in to the EP assessment systems design, those who 
implement this system in the future can achieve similar results. 
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Limitations 

The nature of this project results in several noteworthy limitations. First, the recommended EP 
solutions were recently completed and are not yet released for use; evaluation comments are based 
on the prototype models and design process steps 9 and 10 are not completed. Second, the stake¬ 
holder perception of the EP solutions is based primarily on informal verbal and observed feedback. 
Although the supervising faculty member requested feedback from key stakeholders of each, the 
design teams did not see the direct feedback. No other formal questionnaires or surveys were given 
to evaluate the EPs from the stakeholder’s perspectives. Third, some limitations of EP capability 
existed due to limitations of the university’s EP system. Further, the use of a single university and 
restriction to engineering curricula is a limitation of transferability of the study. Finally, it is acknowl¬ 
edged that the impetus for conducting this work was for meeting a course requirement, although 
the collaborative authorship of this manuscript was an independent, post-course project. 

Potential Future Work 

Lessons learned from this project will not stop with the writing of this manuscript. As the EP 
systems discussed in this paper are used, an end user survey on the EPs strengths, weaknesses, and 
opportunities for improvement is being distributed. In addition, an evaluation of the user’s percep¬ 
tion of the value of EPs prior to and after using these systems can add insight into the importance 
of the EP’s characteristics. Throughout the paper an emphasis on the value and flexibility of EPs 
as an assessment system is discussed and a design process for developing a tailored EP system 
is provided. The information presented herein can be used by others to develop EPs that improve 
graduate student learning and graduate student learning assessment. 
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APPENDIX A 


Case 1 Artifact Collection Matrix Used for Traditional Engineering Department 


Year 1 Year 2 ... Other 

Course Performance 
Technical Papers/Posters/ Presentations 
Degree Milestones 
Additional Accomplishments 
Annual Report 


Case 2 Artifact Collection Matrix Used for Developing Engineering Department 



Graduate School Outcome 

Year 1 

Year 2 

Home Page 

Communications 



Research Reflections 

Research Ability 
Communications 

Technical Literature 



Research Goals 

Research Ability 
Communications 



Publications and 
Presentations 

Communications 

Technical Literature 

Technical Competence 



Plan of Study 

Technical Competence 



Degree Data 




Evaluation and Feedback 
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Case 3 Artifact Collection Matrix Used for Traditional Engineering Research Laboratory 

Year 1 Year 2 


Goals/Feedback/Evaluation 

Program Requirements (Technical Competence) 

Plan of Study 
Required Coursework 
Additional Bio Courses 
Qualifying Exam 
Prelim!Dissertation Proposal 
Final Defense 
Communication Skills 
Resume 

Development of Presentation and Technical Writing Skills 

Instruction/Teaching Experiences 

Seminars Given 

Conference Presentations 

Other 

Research Ability and Scholarly Contributions 

Committee Meetings 
Papers, Submitted 
Journal Publications 
Thesis/Dissertation 
Other 

Awareness and Synthesis of Technical Knowledge 

Seminars Attended 
Papers!Journals Reviewed 
Literature Reviewed 
Proposal Writing Experiences 
Technical Reports 
Other 

Lab Core Requirements 

Safety Requirements 
Intellectual Property Training 
Export Control Workshop 
NSF/NIH Research Integrity Workshop 
Other 

Additional Misc 

Outreach 

Professional Service 
Univ. College. Dept Service 
Lab Leadership 








Awards 

Other 
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APPENDIX B 


Classification 

Traditional Department 

Developing Department 

Research Laboratory 

Objective/Goal 

Develop an EP to meet general 
graduate school outcomes while 
allowing for departmental needs 
to be achieved. Focus on creating 
a system that is effective and 
efficient. No addition al work 
required. 

Assess student understanding 
of the research process 
while providing avenues for 
reflection and faculty feedback. 
Demonstrate compliance with 
graduate school outcomes by 
streamlining and expediting the 
evaluation of student progress. 

Create an assessment system 
to track student progress while 
assessing the quality of work, 
providing timely and accurate 
feedback, recruiting new students 
and providing current students 
with a system to showcase their 
work. 

Basis of Final EP 
System 

Graduate school outcomes 
with the option of expansion to 
individual student goals. 

Stakeholder goals mapped 
to general graduate school 
outcomes. 

The collection of unique and 
specific artifacts. 

EP Value to the 
Client 

Fast tracked system for 
collecting data through direct 
access to student work used for 
departmental accreditation and 
reporting. 

System designed to improve 
overall graduate education 
within the department. 

Electronic data collection for 
departmental reports. 

Laboratory consists of one faculty 
member who chose the system 
to further student education and 
research. 

EP Value to the 
Students 

Due to the changed reporting 
system, students struggled to find 
the direct benefit of the system 
besides departmental gains. 

Students were involved in the 
design process and directly 
connected to the value through 
their involvement. Some were 
cautious due to the added time 
commitment. 

Direct value of the system was 
presented to the students by the 
faculty advisor through concrete 
real world examples. The close 
community of the lab helped to 
raise value. 
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